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Abstract 

We have investigated how uncertainties in the estimation of the detection 
efficiency affect the 90% confidence intervals in the unified approach for con- 
structing confidence intervals. The study has been conducted for experiments 
where the number of detected events is large and can be described by a Gaus- 
sian probability density function. We also assume the detection efficiency has 
a Gaussian probability density and study the range of the relative uncertain- 
ties <7 e between and 30%. We find that the confidence intervals provide 
proper coverage over a wide signal range and increase smoothly and con- 
tinuously from the intervals that ignore scale uncertainties with a quadratic 
dependence on a e . 

Key words: Sensitivity, Systematic uncertainties, Statistical methods, 
Unified approach, Confidence intervals 
PACS: 02.50.-r, 29.90. +r 

1. Introduction 

In the search for new physics, experiments select a region of interest, the 
"signal region," where new phenomena are expected. The search consists 
of selecting a number of events with characteristics of the expected signal, 
but the selection process usually has a finite detection efficiency and includes 
events from uninteresting known sources, generically called background. In 
the absence of a significant excess in the detected number of events over the 
expected background, we use the data to set limits on the range of the pos- 
sible signal at a certain confidence level. The confidence intervals depend 
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not only on the number of detected events, but also on the precision with 
which one knows the detection efficiency and the average background rate in 
the signal region. Therefore, the uncertainties in these quantities will affect 
the sensitivity with which limits can be set by a particular measurement. 
The efficiency and background level are examples of what are referred to 
in the literature as nuisance parameters, and the precision with which they 
are known are measures of systematic uncertainties that degrade the sen- 
sitivity with which the experiment can set limits on the signal of interest. 
For the present study, we do not address how the efficiency or backgrounds 
are obtained, but we assume that estimates for these quantities and their 
uncertainties have been determined for known experimental conditions. 

The procedures for including systematic uncertainties in the determina- 
tion of confidence intervals is a topic of intense interest and discussion. For 
a recent review of the treatment of systematic errors, see Ref. and for a 
review of the treatment of nuisance parameters specifically, see Ref. jsj and 
references therein. There are two primary methods that have been used 
to incorporate systematic errors into the calculation of upper limits. The 
first makes use of the profile likelihood approach (see for example Ref. j3|) 
in combination with the "In £ + | method," which extracts the confidence 
interval by finding the points where the —2 log likelihood function increases 
by 2.706 (90% confidence level). The profile likelihood is evaluated for val- 
ues of the parameters of interest that maximize the likelihood for a specific 
hypothesis for the signal rate and over the entire range of all other nuisance 
parameters of relevance. The second method, which we adopt in this study, 
conducts a full construction of the confidence interval after folding the pri- 
mary probability distribution function with a distribution that describes the 
nuisance parameters. A consensus on which methods are optimal in specific 
experimental situations has not been reached, although the field has matured 
considerably over the past decade. 

This paper focuses on experiments where the number of detected events is 
large and can be described using a Gaussian probability density function. To 
date, the studies for inclusion of systematic uncertainties in the determination 
of confidence regions has concentrated on experiments that record a small 
number of events and whose probability density functions are described by 
Poisson distributions jl, 0]. The case where a large number of events are 
detected in the signal region, consistent with expectations for background, 
is a simpler problem but has received reduced attention. In this case, the 
distribution for the true number of excess events above background fi is 
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Gaussian but physically bounded to be positive. Of course, the number of 
measured events relative to background x can fluctuate above and below 
the expected background level and can be both positive and negative. In 
principle, this situation is a straightforward application of the case for Poisson 
statistics. However, the blind use of those procedures can lead to intense 
computation and is often compounded by issues of numerical precision. 

We construct confidence regions following the approach developed by 
Cousins and collaborators and refined by many others. The method follows 
the frequentist method of constructing confidence intervals proposed by Feld- 
man and Cousins in Ref. p, but it is modified to include systematic errors 
following the original suggestion by Highland and Cousins in Ref. [7fl . There 
are two great advantages of using this unified method for construction of con- 
fidence regions. First, it avoids unphysical regions which must be included 
to interpret classical confidence belts. Second, it has a smooth transition 
between a central confidence belt and upper limits. This has the advantage 
that it defines a priori the choice of the confidence interval, independent of 
measured data. In the absence of systematic errors, the Feldman- Cousins 
prescription coincides with the classical Gaussian central confidence belt for 
positive measurements far from zero. Below that there is a smooth transition 
to single-sided upper limits, corresponding to confidence intervals that are 
non-zero for all values of the measurement x. 

This Neyman construction of confidence intervals is classical or frequen- 
tist, but the systematic uncertainties are included via a Bayesian procedure. 
Systematics are incorporated into the construction by weighting the proba- 
bility distribution with an assumed density for the nuisance parameter (in 
our case, the background level and the detection efficiency) and then inte- 
grating over the nuisance parameter. We assume the nuisance parameters 
have Gaussian probability densities. Integration over the nuisance param- 
eters is Bayesian as it incorporates information about our belief about the 
detection efficiency and the background quantities. 

2. Probability distribution 

For an experiment with low statistics, the confidence limits are a function 
of both the number of observed events (signal plus background) and the 
level of expected background. The correct statistical behavior depends on 
both quantities, since the difference of two Poisson distributions is no longer 
a Poisson distribution. In the limit of Gaussian statistics, the confidence 
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region is determined only by the excess of events over background relative 
to its error, which is dimensionless. This is because the distribution of the 
measured signal (difference between the number of observed events and the 
estimated background) is also a Gaussian distribution with the width given 
by the quadrature sum of the error of the observed events and the error in 
the estimated background level. The probability distribution for the number 
of observed events X with statistical uncertainty ax representing a fixed but 
unknown signal fi and a background level b is given by 



P(X, b, /I, /it,) 



1 



(X-b- fi) 2 



exp 



2a\ 



exp 



(b - Hbf 
2 < ' 



where \ib is the estimated value for b and has an uncertainty a^ b . The prob- 
ability density for the number of signal events x = X — b, or excess above 
background, is then obtained by integrating over the assumed distribution of 
the true background level b: 



P(x,/j,) 



P(X, b, fi, fib) db 

(x - fi) 2 



exp 



2a 5 



(2) 
(3) 



where a = y 1 a 2 x + a 2 b is the statistical uncertainty associated with the ex- 
traction of the signal. 

If we include the uncertainty as in the efficiency S for detecting the signal, 
the distribution is given by the following: 



P{x,fi) 



(X 
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exp 
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(S-S) 2 ' 


exp 


2a 2 



dS, (4) 



where S is the measured detection efficiency, and the integral is performed 
over the posteriori Gaussian probability for the true efficiencies S. We note 
that the integration limits lead to artificial boundaries, which are discussed 
below. We will also refer to S as the "scale factor" for the signal. Defining 
the relative uncertainty as a t = a s /S, the probability distribution is given 
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by 
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Convery [8j derived the expression on the first line of Eq.0 which is valid 
for small values of a e . The more exact expression requires the expression in 
curly brackets containing the difference of the two error functions and was 
obtained by Stenson [9| . We note that the probability distribution converges 
to Eq.[3]when // is very small. This is reasonable, since one expects the prob- 
ability distribution to be unaffected by the scale uncertainty when //=0. The 
probability distributions for several values of // and fixed values of the other 
parameters are plotted in Fig.[TJ The probability distribution proper must 
be normalized by setting the integral to unity, and this is done numerically 
when confidence ranges are calculated. However, for ease of plotting and 
comparing the functions for different parameters in the figures, we normalize 
them by setting P(fi) = 1. From the plots, it is clear that Convery's origi- 
nal expression is a very good approximation over the range of parameters in 
the figures, as it deviates slightly from the full distribution by fractions of a 
percent only for cr e greater than 20%. 

We make some brief remarks concerning the probability function. The 
distribution is a function of the reduced variables x/a and ///cr, the fractional 
scale uncertainty a e and the scale itself S. The first Erf term affects the tails 
of the distribution and becomes increasingly important as a e increases. The 
scale dependence is contained solely in the second Erf term, which evaluates 
to —1 when S is small, generally less than 20%. The origin of the scale de- 
pendence comes from non-physical conditions that occur when the Gaussian 
distribution for S violate the condition < S < 1, so they are of little prac- 
tical interest. In our examples, we have used S = 0.004, which is a typical 
value for the acceptance of some of our experimental searches. However, the 
particular value is unimportant, because the probability distribution is inde- 
pendent of S in our parameter space, and we drop its dependence from the 
argument list. 
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3. Ordering principle 



The integral of the probability density for a = 0.1 defines a 90% confi- 
dence range which is given by 



where, from now on, we assume that x and \i are expressed in units of a. The 
condition on x\ and x% in Eq.[6] is satisfied for many pairs (xi,^), and we 
select a particular pair using the prescription from Feldman and Cousins in 
Ref. 0]. The selection is based on the ratio R of the probability of measuring 
x, given the true mean value /i and scale uncertainty cr e , and the probability of 
obtaining x given the best-fit physically allowable mean value /z=max(O,a;)0: 



The effect of using this ordering principle is to select central confidence in- 
tervals for large values of x and transitioning smoothly to upper confidence 
limits when x is small or negative. The function R(x) is a simple Gaussian 
for large values of /i, but it becomes increasingly asymmetric as /i approaches 
zero. This effect is illustrated for \i = 0.2 in Fig.[2] , where R(x) is plotted 
for four different values of the scale uncertainty a e . 

The pairs (xi, X2) which satisfy both Eq.[6]and Eq.0 define the upper and 
lower limits at the 90% confidence level for a given true mean /1 and scale 
uncertainty a e . For the case considered by Feldman and Cousins in Ref. 0], 
which ignores scale uncertainties, the integrals in Eq.[H] can be specified us- 
ing the standard error function (and its inverse); the condition in Eq.[7] is 
satisfied by requiring ln(i?(xi)) = \n(R(x2)) [lo|. This leads to the following 
conditions: 



lr The value of \x corresponding to the maximum of the probability density actually has 
a weak dependence on the nuisance parameters, but it is ignored here. 






(7) 



x-i — 2/i — Xx, x\ > 



= /i + a//x 2 — 2xi/i, X\ < 0. 



(8) 
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However, given the complexity of Eq.Ofor non-zero a t , the integrals to de- 
termine the confidence intervals, the computation of the ordering principle 
in Eq.[7]as well as the the simultaneous solution for pairs (xi, X2) that satisfy 
both conditions have been evaluated numerically 



4. Evaluation of confidence intervals 

The numerical computation of upper and lower limits was accomplished 



by using a series of ROOT scripts that are described in Ref. [111 ]. The limits 
are computed for a fixed value of the nuisance scale parameter S and for fixed 
values of the assumed relative scale uncertainty cr e . As discussed previously, 
the calculations used the fixed value of 5'=0.004 but are independent of the 
value of this parameter in the region of interest. The Neyman constructions 
of the confidence intervals were computed at fixed values of the mean fi/a 
in increments of 0.1. The range of the measurement x/a was from -5 to 95 
with 1000 bins, corresponding to a granularity of 0.1. The limits are reported 
over the range oix/a from -5 to 45. The additional range of the computation 
was necessary to insure that the normalization integrals fully contained the 
probability distribution function. The resulting limits are shown in Fig. [3J 
Fig.[3]a shows the full range for the calculation of the limits, and [3]b plots 
the limits in the region around x = 0, where the prescription of the ordering 
principle of Feldman and Cousins deviates from the classical central intervals. 
Tablesd] and [2] tabulate the upper and lower limits for x between —3 and 
+3. Also plotted is the result of the limits from the standard Feldman and 
Cousins procedure, which ignores scale uncertainties altogether. We see that 
our calculations converge to the standard prescription for small values of the 
scale uncertainty, as one would expect. However, this result is not guaranteed 
and is an attractive feature of the present calculation. 



5. Coverage 

In a frequentist, or classical approach, coverage is one of the defining char- 
acteristics of a method for determining confidence limits. An ideal method 
will have correct coverage. In the case of 90% confidence limits, a method 
with correct coverage will return limits which contain the true value of the 
parameter of interest in precisely 90% of many repeated trials or experiments. 
Neither overcoverage nor undercoverage are desirable for the method. Over- 
coverage refers to intervals that contain the true value more often than 90%, 
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Table 1: Table of 90% confidence intervals for several values of the relative scale uncertainty 
er c . The second column contains the results of our calculation of the standard Feldman- 
Cousins analysis, which differs by a couple of percent from Table X in Ref. [6(, presumably 
due to numerical approximations. All values are in units of the standard deviation a. 
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Table 2: Table of 90% confidence intervals for several values of the relative scale uncertainty 
er c . The second column contains the results of our calculation of the standard Feldman- 
Cousins analysis, which differs by a couple of percent from Table X in Ref. jf|, presumably 
due to numerical approximations. All values are in units of the standard deviation a. 
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and undercoverage refers to intervals that contain the true value less often. 
A general feeling among physicists is that "erring on the side of caution" (i.e. 
overcoverage) is acceptable, but in this case, the sensitivity of a particular 
experiment is not being fully exploited. Undercoverage, on the other hand, 
can lead to incorrectly counting a fluctuation as a true signal, which is clearly 
an undesirable outcome. 

The coverage provided by the confidence intervals in our approach needs 
to be checked empirically. We note that the standard Feldman- Cousins 
method for determining confidence intervals is known to have overcoverage 
when applied to discrete distributions. This is due to the fact that the inter- 
vals are enlarged to contain an integer number of counts, and this adjustment 
always increases coverage. For our case, where we consider only continuous 
variables fi and x, this is not a consideration. But more to the point, the 
treatment of systematic uncertainties could change the coverage, and in the 



Poisson case |5|, [12J], has also shown to produce overcoverage, typically be- 
tween 0.92 and 0.94. However, the coverage does depend on how one treats 
the nuisance variables. We show that as long as the "true" efficiency S 
and background level fib are treated as random variables, the generated in- 
tervals will have correct coverage over a large range of inputs. In order to 
determine the coverage, we consider an ensemble of repeated Monte Carlo 
"experiments" performed for fixed values of the true parameter /i, the "ex- 
perimental" unbiased estimator for the background level fib, the measured 
detection efficiency S and the relative scale uncertainty a e . For clarity in 
this section, we drop back to the notation from Section [2j where the uncer- 
tainties and scaling variables are shown explicitly. The background level fib 
is assumed to be known from other parasitic experiments or calibrations, or 
to have been calculated using theoretical input. Its uncertainty must also 
be known or estimated. In our numerical examples, we take a^ b = ^fjxbjh. 
Specifically, the following procedure was followed: 

• The true efficiency S was randomly selected from a Gaussian distribu- 
tion G(S, S <r e ), with mean S and a relative uncertainty a e . 

• The "signal" x was then selected from a Gaussian distribution G(Sfi/S, ^JJ^). 

• The true background level b was selected from a Gaussian distribution 

• The observed "background" Xb was selected from a Gaussian distribu- 
tion G{b, sfjlb). 
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• The reduced variable y = (x + Xb — /i&)/cr y , where a y = ^Jx + Xb + cr 2 b , 
was used to determined the 90% CL for the specified value of a t , re- 
turning the lower and upper limits y L and y u . 

• If the true value \i was contained in the confidence interval [y L <J y , y u &y\, 
it was counted; otherwise, it was not. 

At the end of a specified number of trials (one million), the coverage was 
calculated as the fraction of times the true value was contained within the 
confidence interval. The results are shown in Fig.H] at four values of the 
reduced background level Hb/cr y for various values of a e . The coverage for 
the case of fib/cr y =0 was determined by generating the variable y directly to 
avoid calculations where a y = \fx was very small. The figure shows that 
correct coverage is achieved within 1% for 0.5 < fi/<T y < 10. For fi/a y less 
than 0.5, there is a systematic increase of the coverage that depends on the 
background level. The increase is due, in part, to numerical approximations 
for large negative values of the measured quantity x, because for small //, 
the procedure itself should converge to the standard Feldman and Cousins 
result, which ignores systematic uncertainties and shows correct coverage. 



Punzi |13| has studied the coverage for various ordering algorithms and has 
shown that an ordering algorithm approximately based on the profile likeli- 
hoods can generate correct coverage for large values of the signal parameter 
practically independent of the scale uncertainty. However, for small values 
of the signal, which is governed by Poisson statistics, the coverage of this 
method approaches unity. 



6. Example 

By way of example, we take a histogram where each bin contains an esti- 
mate for the number of events above background as a function of a variable 
that represents, for example, the kinematic coverage of the experiment. The 
number of signal events is negative when the estimated background exceeds 
the number of measured events, and the errors on each point include the sta- 
tistical uncertainties of the unsubtracted data as well as the uncertainty in 
the estimate of the mean background level (recall Eq.[3]). The same data with 
its errors is shown in Fig. [5], together with lower and upper limits determined 
with our method for four different assumptions for the relative systematic 
uncertainty (<r e =0.03, 0.10, 0.20 and 0.30). The plots provide a picture of 
how the systematic uncertainties might change the limits that can be set 
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using a given data set. The limits on the first plot are very close to what 
would be obtained with the standard Feldman and Cousins analysis. As the 
systematic error increases, so do the upper limits. 



7. Dependence of limits on the scale uncertainties 

The application of our procedure for determining upper limits that we 
have described so far is quite cumbersome. Many steps of computation are 
required to calculate the confidence intervals, and even once determined, the 
limits are only valid for a fixed value of the relative scale uncertainty. If limits 
are to be determined for a fixed value of <r e , as in the previous example, 
the procedure is straightforward. However, it would not be unusual if a e 
depends, for example, on the kinematics of the experiment, which would add 
considerable complexity to the entire procedure. Therefore, to streamline 
the calculations, we have investigated, and found, a simple dependence of 
the limits on systematic uncertainties. 

The dependence of upper limits on the scale uncertainty is of interest 
in its own right and was investigated in the original study by Cousins and 
Highland in Ref. 0. In that low-counting application, they concluded that 
the upper limit U with a systematic uncertainty a e can be calculated from 
the limit Uq with no systematic error using the following formula: 

U = U (l + aU a 2 £ /2), (9) 

with a =1. Motivated by this study, we have fitted our results to Eq.[9j 
For the limited range of x < 3, shown in Fig.0 our results are described 
quite well with the empirical value of a=2 if we substitute the standard 
Feldman and Cousins result for Uq. We also note that this procedure has 
the desirable property of continuity, that is it reduces to the Feldman and 
Cousins result for a e =0. For reference, we show the dependence obtained 
by Rolke and collaborators in Ref. 0] and by Punzi 13] normalized to our 



result at x = 1, which roughly correspond to the parameters for their study. 
The upper limits from Rolke show a dependence consistent with a stronger 
dependence on the scale uncertainty with a = 6, but the data from Punzi, 
except for their the last point, is in very good agreement with our analysis. 
It must be noted, however, that these comparisons can only be qualitative, 
since their applications assume Poisson statistics. 

In order to cover a larger range of x (x <10), it is necessary to extend the 
parameterization in Eq.[H]by allowing a to be a function of x. Happily, the 
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following common parameterization can be used for both upper and lower 
limits: 



a(x) 
a(x) 



Aexp(-Bx/2) 
A exp(-B) 



x < 2, 



x > 2 



(10) 



with Ay = 2.26 and Bjj = 0.092 assigned for upper limits, and A L = —1.34 
and Bl = 0.134 used to determine the lower limits. Using this prescription, 
we compare our calculations for <7 e =0.3 with the scaled Feldman- Cousins 
result from Ref. 0] using Eq.[9j There is good agreement as shown in Fig.[71 
We also note that the lower limit is relatively insensitive to the systematic 
error for x less than 3. However, the upper limit begins deviating from the 
standard result for x greater than O.The deviations of the scaling from the 
calculations are typically within the numerical accuracy of our calculations 
(0.1 a), but they can be as large as 0.4a at some extremes. 

8. Summary and conclusions 

We have investigated how uncertainties in the estimation of the back- 
ground levels and detection efficiency affect the 90% confidence intervals 
in the unified approach of Feldman and Cousins from Ref. pj. The study 
has been conducted for experiments where the number of detected events 
is large and can be described by a Gaussian probability density function. 
The construction of confidence intervals is classical or frequentist, but the 
systematic uncertainties are incorporated into the construction by weighting 
the probability distribution with an assumed density for the nuisance pa- 
rameter and then integrating over the nuisance parameter. This integration 
adds a Bayesian flavor to the approach as it incorporates information about 
our belief about the detection efficiency and the background quantities. We 
assume the nuisance parameters have Gaussian probability densities, and 
have studied the effect of relative scale uncertainties in the range between 
< a t < 0.3. 

We find that the 90% confidence intervals increase quadratically with 
the size of the relative scale uncertainties (Eq.[9]), as anticipated from previ- 
ous work. The confidence intervals that incorporate scale uncertainties can 
be obtained from the unified confidence intervals proposed by Feldman and 
Cousins in Ref. with a simple scaling algorithm. The new intervals have 
several attractive features. For small values of the true mean fi, the scale 
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uncertainties do not affect the confidence intervals, as one might expect. By 
construction, the confidence intervals should have correct coverage, and this 
has been verified by Monte Carlo calculations. Finally, in the limit that a e 
approaches 0, the intervals reproduce the intervals by Feldman and Cousins 
that ignore systematic uncertainties. 
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Figure 1: (Color online) The probability distribution is plotted for a t =0.3 and 5=0.004 
and four different values of the true mean ji. The exact solution by Stenson 0] is shown 
(solid line) along with the approximate solution by Convery Q (dashed line). The function 
is approximately Gaussian with a modified width for all values of the parameters. The 
curves practically coincide in the top plots, and the integrals of the curves are compared 
for two different intervals, showing that the approximate solution is very good. 
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a) b) 




X X 



c) d) 




X X 

Figure 2: (Color online) The ordering principle R is plotted for four different values of 
the relative scale uncertainty, <r e , at fj,— 0.2. The function R develops a tail at low x as a e 
increases. The tail has the effect of biasing the selection of the confidence interval toward 
a one-sided confidence interval. At this low value of [i, the tail is very shallow and extends 
to very large negative x. 
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Figure 3: (Color online) The 90% CL confidence intervals are plotted as a function of 
the measurement x. Both axes are plotted in units of the standard deviation of the 
measurement a. The four limits correspond to relative scale uncertainties of cr e = 0.03, 0.1, 
0.2 and 0.3, respectively. The black curves show the standard Feldman-Cousins analysis, 
which ignores systematic uncertainties altogether, a) Limits plotted on a large scale, b) 
Limits plotted on a finer scale around x = 0. For negative values of x, the curves converge 
to the standard Feldman-Cousins analysis. 
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Figure 4: (Color online) The computed coverage for 90% confidence limits is shown for 
various values of the scale uncertainty a e at four values of the background level. The 
titles emphasize that variables are measured relative to the uncertainty (see text for de- 
tails). Most of the scatter about 0.90 is likely due to numerical approximations in the 
determination of the confidence ranges. However, for small values of fJ>/cr y , the coverage 
should approach the value given by the standard Feldman and Cousin result (shown as 
solid squares) but in fact shows a systematic increase which is somewhat dependent on 
the background level. 
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Figure 5: (Color online) The upper and lower limits computed at the 90% CL using four 
different assumptions for the relative scale uncertainty a c for the same sample of data a) 
er e =0.03, b) cr e =0.10, c) <7 e =0.20, and d) t7 e =0.30. The limits in a) are very close to those 
obtained using the standard Feldman and Cousins analysis. 
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Figure 6: (Color online) The 90% upper limit is plotted as a function of the relative 
scale uncertainty a e for various values of the measurement x. The standard Feldman 
and Cousins result, which ignores systematic uncertainties, is plotted for a e =0. The 
dependence of the upper limit on the scale uncertainty computed by Rolke et. al. Q 
(open circles) and Punzi [l3| (open squares) are also shown normalized to our upper limit 
for x = 1. The data from Ref. [13], excluding the last point, are consistent with our 
analysis. 
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Figure 7: (Color online) Three sets of confidence intervals are plotted. The central curves 
(open circles) reproduce the result of Feldman and Cousins from Ref. which ignores 
systematic errors. The curves slightly inside the outer curves (open squares) are the 
calculated limits assuming a relative scale uncertainty of <r e =0.3. The outer curves (solid 
squares) are scaled from the Feldman-Cousins result using Eq.[9] The scaling provides a 
very good representation of the calculations, but deviates slightly at x ~ 10. 
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